Skip to content

F# hot reload: Edit-and-Continue delta emission behind --test:HotReloadDeltas#19941

Draft
NatElkins wants to merge 160 commits into
dotnet:mainfrom
NatElkins:hot-reload-v2
Draft

F# hot reload: Edit-and-Continue delta emission behind --test:HotReloadDeltas#19941
NatElkins wants to merge 160 commits into
dotnet:mainfrom
NatElkins:hot-reload-v2

Conversation

@NatElkins

@NatElkins NatElkins commented Jun 12, 2026

Copy link
Copy Markdown
Contributor

This draft PR presents the complete F# hot reload implementation as a single branch against current main: dotnet watch (with a companion dotnet-watch change, linked below) patches running F# processes in place via the standard EnC pipeline (MetadataUpdater.ApplyUpdate) — the same runtime contract C# uses. Unsupported edits degrade to the rebuild-and-restart flow watch uses today, so a limited scope still behaves as a complete feature.

It is opened as a single draft per discussion with @T-Gro: one branch to review, build, and launch testing from. A decomposition into independently shippable PRs is laid out below and can begin whenever review reaches that stage. Earlier discussion: #11636.

Try it (30–45 min, two clones, copy-paste steps)

docs/hot-reload-quickstart.md walks from git clone to editing a running F# app — including adding a List.map (fun s -> s.ToUpper()) to a live method and watching it apply in place, state preserved. Short version: build this branch, build NatElkins/sdk fsharp-hotreload-watch-v2 (a complete .NET CLI with an F#-aware dotnet watch), copy the freshly built FSharp.Compiler.Service.dll into the SDK layout, add <OtherFlags>$(OtherFlags) --test:HotReloadDeltas</OtherFlags> to a console app, and dotnet watch run.

What works

  • Method body edits, including bodies containing closures, async, resume-point-stable task, and generics
  • Resumable state-machine shape edits (behind --test:HotReloadClassStateMachines): adding or removing a let!/do! in task and backgroundTask, which emits class-form (reference-type) state machines so the change is an AddInstanceFieldToExistingType plus a method update, matching C#. taskSeq and other resumable CEs share the same lowering path. Off by default; flag-off codegen is byte-identical.
  • Lambdas added / edited / removed across generations (new closure classes synthesized into the running process)
  • Member additions: methods, module functions and values, static/instance fields, properties, [<CLIEvent>] events
  • New type definitions: classes, records, unions, structs, modules, enums, interfaces, delegates, units of measure
  • Attribute add/change/remove on existing members; parameter renames
  • Multi-project: one watch session, per-project baselines, interleaved edits across an app and its referenced libraries
  • Line-shift edits (comment/whitespace above code) become pure sequence-point line updates — no delta, no restart
  • Rude edits (signature changes, captured-variable rename/type/scope changes, plus the F#-specific inline-annotation change) degrade to the standard rebuild-and-restart flow with a precise diagnostic. Adding or removing a mid-sequence let!/do! is rude under the default struct state machines, but becomes a supported edit (an AddInstanceFieldToExistingType plus a method update, matching C#) under --test:HotReloadClassStateMachines

Isolation and disabling

The feature is designed so that a compile without the flag is indistinguishable from main, and so that the whole feature can be turned off at one point if something goes wrong:

  • Off by default. Everything is behind --test:HotReloadDeltas (requires --debug+, incompatible with --optimize+), with class-form resumable state machines a further opt-in behind --test:HotReloadClassStateMachines. Both are off by default; no MSBuild property or SDK behavior changes in this repo.
  • Flag-off output is byte-identical to main, pinned by the EmittedIL suite (1212 baseline tests) and by dedicated determinism tests (DLL+PDB byte-equality across recompiles and across graph/sequential checking modes).
  • Flag-off cost is zero beyond cheap checks. A dedicated audit pass removed all unconditional work from the flag-off path: no reflection, no eager metadata snapshots, no per-name or per-closure side-channel probes, no extended lifetime of the optimized typed tree, and FSharpProjectSnapshot.fs is byte-identical to main (tracked-input staleness lives in the hot reload session layer instead).
  • One seam. The compiler driver integration is a single ICompilerEmitHook interface with a no-op default. Guard scripts under tests/scripts/ (run by the verification gate) enforce that only the intended files consume the hook and that the IlxGen name-generation path and fsi surface stay on their pinned shapes.
  • Disabling: drop the flag (compiler side); the companion dotnet-watch side additionally has a one-variable kill switch (DOTNET_WATCH_FSHARP_HOTRELOAD=0) that restores stock restart-on-edit behavior.

Architecture

  • Sessions are an explicit entity: FSharpChecker.CreateHotReloadSession returns a FSharpHotReloadSession holding per-project committed snapshots + emit baselines — the DebuggingSession/CommittedSolution shape from Roslyn, built on FSharpProjectSnapshot (the snapshot contract, not the experimental workspace surface), so it composes with the FSharpWorkspace direction without depending on it. Solution-wide commit/discard semantics, runtime-capability updates, active-statement intake.
  • A typed-tree semantic diff classifies every edit (Roslyn's SemanticEdit/RudeEdit model), gated on the runtime's advertised EnC capabilities (AddMethodToExistingType, NewTypeDefinition, GenericUpdateMethod, …). Anything unclassifiable fails closed.
  • Delta emission produces the standard EnC triplet (metadata/IL/PDB deltas with EncLog/EncMap), validated against recorded Roslyn EmitDifference reference deltas, with mdv, and against CoreCLR ApplyUpdate in runtime tests.
  • Closure identity is solved the way Roslyn solves it, adapted to F#'s lowering: lambdas get stable identity from a typed-tree occurrence model (ordinal chains + LCS alignment rather than syntax offsets), persisted in Roslyn's exact portable-PDB EnC CDI blob formats (the encoder round-trips Roslyn's own blobs byte-identically), with deterministic occurrence-derived closure-class names so any process can reconstruct identity from the PDB alone.

Design documentation (rendered, on this branch)

Doc What it covers
hot-reload-architecture.md Start here. The entity model: FSharpHotReloadSession, per-project committed snapshots + baselines on FSharpProjectSnapshot, the FSharpWorkspace relationship, determinism pins
hot-reload-closure-mapping.md The closure problem and its solution: lambda occurrence model, Roslyn-format EnC CDI PDB blobs, occurrence-derived deterministic closure naming, cross-process reconstruction; state-machine handling
hot-reload-member-additions.md Recorded Roslyn EmitDifference reference templates (EncLog/EncMap shapes per edit kind) and the F# emission matrix incl. every intentional fail-closed case
hot-reload-capabilities.md Runtime capability negotiation (Roslyn EditAndContinueCapabilities parity) and per-capability gating
hot-reload-active-statements.md The debugger-contract mirror: active statements, sequence-point updates, remapping; host wiring deferred

Scale and review shape

~118 commits, 159 files, ~60k insertions, of which ~34k are tests and ~2.4k docs. The src/ changes are predominantly new self-contained modules (IlxDeltaEmitter.fs, TypedTreeDiff.fs, HotReloadBaseline.fs, the AbstractIL EnC readers, the delta writer stack). Pre-existing files carry hook callouts plus one behavior-neutral refactor (ilwrite.fs MetadataTable record→class, exposing a baseline-row access seam for the delta writer); total deletions across the branch are 241 lines.

Proposed path to merging in pieces

The fail-closed design means scope can grow capability by capability — each unsupported case is already a rude edit with a diagnostic, so every intermediate state is a complete, working feature. The natural sequence:

  1. Behavior-neutral AbstractIL/ilwrite foundations (no feature, byte-identity evidence) — already staged separately as a 3-commit branch (refactor: AbstractIL EnC foundations (hot reload stack 1/n) NatElkins/fsharp#2)
  2. Session entity + typed-tree classification, with every edit classified rude — end-to-end complete at minimal scope (watch restarts on every edit, but through the proper pipeline). This is the PR where the architecture gets decided with reviewers.
  3. Method-body-only deltas (the core)
  4. Closure mapping + deterministic naming (the one chunk that touches IlxGen/name-generation paths — reviewed on its own)
  5. Member/field/type additions, generics, state machines — each already individually capability-gated
  6. Active statements / sequence-point updates

Evidence

Known limitations / future work

Companion PRs

claude and others added 30 commits June 10, 2026 14:27
…eaps, row indexing, EncLog writer)

Ported verbatim from the hot-reload prototype branch onto current main:
- ILDeltaHandles.fs: typed handles for EnC delta metadata rows
- ILMetadataHeaps.fs: heap accounting shared by baseline and delta passes
- ILRowIndexing.fs: row index helpers for delta table emission
- ILEncLogWriter.fs: IEncLogWriter abstraction recording EncLog/EncMap entries
- ilbinary.fs: additive table/coded-index definitions required by EnC emission
- EnvironmentHelpers.fs: FSHARP_HOTRELOAD_* environment variable helpers
- Caches.fs: qualify System.Guid to avoid shadowing from new helpers

All additions are dormant without the hot reload flag; no behavior change
for normal compilation.
…ble name-map state

- Generated/GeneratedNames.fs: central helpers for synthesized member/type names
- Generated/CompilerGeneratedNameMapState.fs: capture/replay of name-generator
  state so successive hot reload generations synthesize identical names
- CompilerGlobalState.fs: thread name-map state through NiceNameGenerator
- IlxGen.fs: route compiler-generated local names through a single freshIlxName
  helper (replaces direct IlxGenNiceNameGenerator.FreshCompilerGeneratedName
  call sites) and add IlxGenEnvSnapshot/snapshotIlxGenEnv/restoreIlxGenEnv to
  capture the codegen environment a delta generation must replay
- IlxGen.fsi: expose the snapshot API

3-way merge against current main verified: all 10 freshIlxName sites intact,
no new upstream name-generator call sites bypass the helper, net diff vs main
matches the prototype footprint exactly (+85/-49).
- TypedTree/SynthesizedTypeMaps.fs: stable maps for synthesized (closure/
  state-machine) types across generations
- TypedTree/TypedTreeDiff.fs: semantic diff between baseline and updated
  typed trees - binding/tycon snapshots, FNV-1a digests, rude-edit
  classification (signature changes, constraint changes, mutable-field
  toggles, type layout changes)
- HotReload/DefinitionMap.fs: maps baseline definitions to updated symbols

Static port validation against current main: every TypedTreeOps symbol
consumed by TypedTreeDiff.fs resolves in the refactored namespace (upstream
split TypedTreeOps.fs into TypedTreeOps.*.fs with [<AutoOpen>] modules
inside namespace FSharp.Compiler.TypedTreeOps, so consumer opens are
unchanged).
- ilwrite.fs/.fsi: thread IEncLogWriter through metadata emission; expose
  ILTokenMappings, MetadataHeapSizes, MetadataSnapshot and
  WriteILBinaryInMemoryWithArtifacts so a compilation can capture the token
  maps and heap layout a later delta generation must build on;
  markerForUnicodeBytes made public for delta #US emission (ECMA-335
  II.24.2.4)
- ilwritepdb.fs: portable PDB hooks for baseline capture
- ILBaselineReader.fs: pure F# reader recovering baseline metadata state
  (row counts, heap sizes, GUID heap start) from emitted images
- HotReloadBaseline.fs: baseline snapshot assembly for hot reload sessions
- HotReloadPdb.fs: PDB delta support shared across generations

3-way merged cleanly against current main; upstream MethodDefKey/Codebuf
changes (compareILTypes etc.) retained. Flag-off emission path is unchanged:
full builds use the no-op EncLog writer (createNullEncLogWriter).
EnC delta (#~, #Strings, #US, #Blob, #GUID) construction, ported verbatim:
- FSharpSymbolChanges.fs: symbol-level change set consumed by the writer
- IlxDeltaStreams.fs: per-generation heap/stream builders (delta-local
  offsets, generation-aligned Blob/US heaps)
- FSharpDefinitionIndex.fs: definition-to-token index over the baseline
- DeltaMetadataEncoding.fs / DeltaMetadataTypes.fs / DeltaMetadataTables.fs:
  ECMA-335 II.22/II.24 row encodings, coded indices, table models
- DeltaTableLayout.fs / DeltaIndexSizing.fs: row layout and wide/narrow
  index sizing for delta images
- DeltaMetadataSerializer.fs: serializes EncLog/EncMap-ordered tables
- DeltaMetadataSrmWriter.fs: System.Reflection.Metadata-based parity writer
  (FSHARP_HOTRELOAD_COMPARE_SRM_METADATA)
- FSharpDeltaMetadataWriter.fs: top-level per-generation metadata delta
  writer (EncId/EncBaseId chaining)
- SymbolMatcher.fs: matches baseline symbols to updated tree symbols
- HotReloadAccessorTypes.fs: synthesized accessor type support
- IlxDeltaEmitter.fs: per-generation IL emission for added/updated methods,
  async/state-machine @hotreload type synthesis, rude-edit guarded
  (HotReloadUnsupportedEditException instead of failwith on unresolvable
  references)
- HotReloadState.fs: per-session mutable state (generation chain, name maps)
- DeltaBuilder.fs: orchestrates diff -> symbol matching -> emission
- HotReloadCapabilities.fs: capability flags (Baseline, AddMethodToExistingType, ...)
- HotReloadContracts.fs: cross-layer contracts for tooling integration
- RudeEditDiagnostics.fs: rude-edit kinds and diagnostic mapping
- EditAndContinueLanguageService.fs: Roslyn-shaped EnC entry point; rejects
  all rude edits before invoking the emitter
- Adapters.fs: adapter layer between FCS session API and EnC service
- CompilerOptions.fs: --enable:hotreloaddeltas (baseline capture emission)
  and --enable:hotreloadhook (synthesized-name replay only)
- CompilerConfig.fs/.fsi: emitCaptureArtifacts config and emit-hook wiring
- CompilerEmitHookState.fs / CompilerEmitHookBootstrap.fs /
  HotReloadEmitHook.fs: pluggable emit hook capturing baseline artifacts
  (token maps, metadata snapshot, PDB) during normal fsc emission
- fsc.fs: invoke the emit hook around main4-main6 binary emission
- fsi.fs: qualify System.Guid (shadowing from new compiler-level opens)
- FSComp.txt: fscHotReloadRequiresDebugInfo (2026),
  fscHotReloadIncompatibleWithOptimization (2027) - IDs verified free on
  current main; xlf regeneration deferred to a follow-up commit (requires
  a build with /p:UpdateXlfOnBuild=true)

Flag-off compilations bypass the hook entirely (null EncLog writer).
- service.fs/.fsi: session surface the IDE/dotnet-watch layer consumes:
  FSharpChecker.StartHotReloadSession (2 overloads), EmitHotReloadDelta
  (2 overloads), EndHotReloadSession, HotReloadSessionActive,
  HotReloadCapabilities; plus FSharpHotReloadError, FSharpHotReloadDelta,
  FSharpAddedOrChangedMethodInfo, FSharpHotReloadCapability/-ies types
- FSharpProjectSnapshot.fs: snapshot support for session baselines
- FSharpCheckerResults.fs: expose typed-tree access needed by the differ
  (sessions require keepAssemblyContents=true)

One textual conflict resolved in the service.fs open block (upstream added
FSharp.Compiler.Caches at the same spot the prototype added
ILDynamicAssemblyWriter/ILPdbWriter; kept all three). Upstream's new
single-argument DiagnosticSink signature is unaffected - ported code
implements no DiagnosticsLogger. Net diff vs main matches the prototype
footprint (+927 service-layer lines).
- tests/FSharp.Compiler.Service.Tests/HotReload/ (18 files): unit tests for
  TypedTreeDiff, delta metadata writer, coded indices, ILBaselineReader,
  portable PDB reader, SRM parity, rude edits, thread safety, generated
  names, error/edge paths (258 tests on the prototype branch)
- tests/FSharp.Compiler.ComponentTests/HotReload/ (15 files): integration
  tests - baseline capture, delta emission, runtime MetadataUpdater
  ApplyUpdate scenarios, mdv validation, PDB generations (101 tests)
- FSharpWorkspace.fs: workspace plumbing used by hot reload session tests
- HotReload.runsettings sets DOTNET_MODIFIABLE_ASSEMBLIES=debug and
  COMPlus_ForceEnc=1 for ApplyUpdate runs; NOTE: repo moved to
  Microsoft.Testing.Platform since the prototype - verify
  RunSettingsFilePath is still honored when running these suites
- mdv-based tests honor FSHARP_HOTRELOAD_MDV_PATH

Test fsproj compile items re-applied onto the MTP-migrated project files
(anchors verified; 3-way merge clean).
…hook, release note

- tests/scripts/: hot-reload-verify.sh (full pipeline gate),
  hot-reload-demo-smoke.sh (HOTRELOAD_SMOKE_RUNTIME_APPLY runtime apply),
  metadata coupling/parity and plugin-boundary guards,
  check-ilxgen-name-path.sh, main-fsi drift checks + allowlists
- tests/projects/HotReloadDemo/: console demo app with hot reload session
  driver (hotreload-session.json + edited DemoTarget.fs); registered in
  FSharp.slnx under /Tests/HotReloadDemo/ (upstream migrated .sln -> .slnx
  since the prototype, so the old FSharp.sln entries were re-expressed)
- eng/Build.ps1: Invoke-HotReloadDemoSmokeTest hook after testCoreClr and
  testDesktop runs, re-derived onto upstream's reworked test sections
  (TestSplit batching); runs the demo app with
  DOTNET_MODIFIABLE_ASSEMBLIES=debug and asserts the delta-emitted marker
- hot-reload-verify.sh updated to build FSharp.slnx
- tools/hot-reload/compare_roslyn.fsx: Roslyn delta comparison helper
- docs: debug-emit.md hot reload heap tracing section,
  hot-reload-tgro-closure-matrix.md, release note entry in 11.0.100.md

Deliberately not ported from the prototype branch: tmp.fsx (scratch),
HOT_RELOAD_REVIEW_CHECKLIST.md and CLAUDE.md (workflow files, not product);
xlf regeneration deferred (needs /p:UpdateXlfOnBuild=true build).
Mirror Roslyn's EditAndContinueCapabilities flags and parser semantics
(exact-name matching, unknown capability words ignored, the
AddDefinitionToExistingType aggregate) with an immutable typed model in
FSharp.Compiler.EditAndContinue. Runtime capability strings are parsed
once at the session boundary; a session always carries at least the
Baseline capability. Includes parser tests and a design doc.
…bilities

Thread the typed capability model through the hot reload session:
FSharpChecker.StartHotReloadSession accepts an optional capabilities
string sequence (Roslyn WatchHotReloadService parity), parsed once and
stored on the session state; when omitted the session is conservative
and assumes baseline-only support.

Method additions now require the AddMethodToExistingType runtime
capability. Without it the diff reports the new
RudeEditKind.NotSupportedByRuntime (FSHRDL016) naming the missing
capability, mirroring Roslyn's distinction between edits unsupported by
hot reload and edits the connected runtime cannot apply. Classification
consults a single capabilityForAddition seam so Phase B can flip field
additions from always-rude to capability-gated by implementing emission.
Rude-edit details now flow into the UnsupportedEdit error so hosts see
the missing capability.
… additions

Module-level values lower to a static field plus accessor methods with
initialization in the startup class constructor, so adding one requires both
AddStaticFieldToExistingType and AddMethodToExistingType; module-level
functions lower to plain static methods and require only the latter. Both
were previously unconditional DeclarationAdded rude edits. Instance fields
remain rude until field-row emission lands (Phase B2); an emission-level test
pins the cut line so a capability-enabled host gets a clean UnsupportedEdit
rather than a half-emitted delta.
Adds Field-table support end-to-end in the delta writer pipeline:
FieldDefinitionRowInfo row model, mirror table population, serializer
table wiring, and SRM shadow-writer parity (including Field in the
tracked parity tables).

EncLog follows the Roslyn pattern recorded from a hotreload-delta-gen
C# reference delta that adds a static field with an initializer: the
parent TypeDef row is logged with the AddField operation immediately
followed by the new Field row with the Default operation, and only the
Field row enters EncMap. The pairs are spliced between the Module entry
and Method entries so per-table EncLog sorting cannot separate them.

Writer-level test asserts the exact EncLog sequence, EncMap content,
and serialized table stream for an added static int32 field.
Added module-level values (let mutable x = ...) now emit complete,
runtime-appliable deltas: the static backing fields on the startup-code
class enter the Field table, accessors and the startup constructor are
emitted as added methods, and the added value is readable/writable via
its accessors after MetadataUpdater.ApplyUpdate.

Making the deltas actually apply required aligning the EncLog with what
CoreCLR's EnC applier (CMiniMdRW::ApplyDelta) and Roslyn emit for ALL
added member kinds: an added member logs its PARENT row tagged with the
Add* operation immediately followed by the member row with the Default
operation (TypeDef for methods/fields, MethodDef for parameters,
PropertyMap/EventMap for properties/events; map rows and MethodSemantics
rows are plain Default entries). The previous shape - the Add* op on the
member row itself - corrupted parent member lists at apply time and had
never been runtime-applied by any test.

Further fixes uncovered by the runtime-apply tests:
- EditAndContinueOperation numeric values corrected to the CLR/SRM codes
  (AddParameter=3, AddProperty=4, AddEvent=5; previously 4/5/6).
- Baseline #Strings size now uses SRM's trimmed size (StringHeap.TrimEnd
  semantics, Roslyn EmitBaseline parity); the padded stream size shifted
  every delta-heap string reference.
- Added member names/signatures are written into the delta heaps instead
  of reusing fresh-compile heap offsets, with signature blobs remapped
  through remapSignatureBlobWith.
- Return-parameter rows are no longer synthesized: the out-of-sequence
  seq-0 rows forced the CLR's indirect ParamPtr path and made ApplyUpdate
  reject the delta. ParamList of added methods stays monotone instead.

DeltaBuilder resolves the startup-code class constructor as an updated
method when the baseline already contains it (it is not a typed-tree
binding, so the diff cannot pair it). Added field tokens chain into the
next-generation baseline like added methods.

Initialization semantics (validated at runtime and documented): the
initializer runs lazily if the startup class was not yet type-initialized
(the test observes 41); an already-initialized type reads default(T).

New tests: checker-level delta validation for the added value, mdv
validation of the Field rows and AddField pairing, and runtime ApplyUpdate
tests for both added module functions and added module values. Instance
field additions remain rejected (Phase B2).
…ule values

Generation 2 adds a second mutable module value on top of the generation-1
field addition and applies it with MetadataUpdater.ApplyUpdate. This pins
baseline chaining: generation-1 field/method/property tokens resolve in the
chained baseline so only the new value is appended, and the startup-code
constructor added in generation 1 is re-emitted as an updated method body.

Also pins the already-initialized regime of the initialization semantics:
the startup class was type-initialized in generation 1, so the second value
reads default(int) rather than its initializer, while generation-1 state is
preserved across the update.
… the typed-tree diff

Replace the blanket lambda-shape digest equality check with a structured
per-member lambda occurrence model:

- Occurrence identity = traversal ordinal + parent-lambda chain + structural
  digest (curried arity, per-group parameter type identities, capture identity
  list, return type identity). Consecutive curried lambdas form one occurrence,
  matching IlxGen closure formation. Source ranges are diagnostics-only.
- Old/new alignment = two LCS passes (full structural digest, then shape-only),
  so inserting/removing/reordering lambdas aligns the surviving occurrences
  instead of reporting spurious removed+added pairs.
- Capture compatibility per matched pair with C#-parity classification:
  RenamingCapturedVariable, ChangingCapturedVariableType,
  ChangingCapturedVariableScope (cross-occurrence move post-pass), plus
  additions/removals (additions may become applicable in C4 via
  AddInstanceFieldToExistingType).
- Classification: pure body edits within an unchanged lambda set remain plain
  MethodBody edits; lambda-set changes stay LambdaShapeChange rude edits, now
  with counts/ordinals in the message and a structured LambdaEdits payload on
  TypedTreeDiffResult for the C4 emitter.
- Quotations, object expressions, local type functions, and types without a
  computable runtime identity keep the legacy whole-body digest path unchanged.

Move the closure-mapping design doc into docs/ with C1 implementation notes.
Add EncMethodDebugInformation (CodeGen), replicating byte for byte the three
Roslyn EnC CustomDebugInformation blob formats persisted per method in
portable PDBs (EnC Local Slot Map, EnC Lambda and Closure Map, EnC State
Machine State Map), including the syntax-offset-baseline optimization and
the slot-map kind/ordinal byte packing. In F#, the syntax-offset slots carry
C1 occurrence keys packed from the occurrence ordinal chain (16-bit
segments, fail-closed past limits).

Tests cover round-trips (temps, ordinal-flagged slots, negative baselines,
static/this-only closure ordinals, negative state numbers), golden bytes,
fail-closed limits, and cross-validation: a C# library built with the repo
SDK has its Roslyn-emitted CDI rows decoded with the new decoder and
re-encoded byte-identically.

PDB writer/reader wiring is the next C2 commit.
… hotreloaddeltas

When --enable:hotreloaddeltas is on, the fsc emit path computes per-method
lambda occurrence data with the Phase-C1 extraction over the same optimized
typed tree the baseline capture snapshots, serializes it with the C2 blob
encoders, and carries it into the portable PDB writer through a new
methodCustomDebugInfoRows side channel on the IL writer options:

- TypedTreeDiff.collectMemberLambdaOccurrences: public C1 extraction over a
  CheckedImplFile, returning every member binding (empty occurrences for
  members the model cannot represent).
- EncMethodDebugInformation.computeMethodCustomDebugInfoRows: occurrences ->
  EnC Lambda and Closure Map blobs keyed by IL method (compiled) name, with
  occurrence keys packed from the C1 ordinal chains; MethodOrdinal stays
  UndefinedMethodOrdinal; one closure scope per occurrence (IlxGen lowers
  every occurrence to its own closure class). The EnC Local Slot Map is
  omitted: the lowered slot layout is an IlxGen artifact, not derivable from
  the typed tree.
- ilwritepdb attaches the rows as CustomDebugInformation on MethodDef parents,
  but only when the method name identifies exactly one method row; the
  producer likewise drops compiled names claimed by more than one member, so
  a map can never attach to the wrong method (overloads fail closed and
  simply carry no map).

Flag-off builds pass the empty map everywhere and emit byte-identical output
(EmittedIL gate: 1212 passed, 0 failed). New PdbCdiEmissionTests verify the
flag-on PDB carries a decodable map with occurrence keys [0] and [0; 1] for a
nested-lambda sample, and that the flag-off PDB contains no EnC CDI rows.
…sion

When a baseline is captured, decode every method-level EnC CustomDebugInformation
row of the baseline portable PDB (lambda/closure map plus the slot and state-machine
maps) into FSharpEmitBaseline.EncMethodDebugInfos, keyed by MethodDef token - the
CDI parent is a MethodDef handle, so token keying is unambiguous, unlike the
name keying on the write side which exists only because the PDB writer lacks
tokens. The fsc emit hook decodes the exact emitted PDB bytes; the checker path
additionally reads the on-disk PDB as a sibling input because its in-memory
baseline rewrite carries no CDI side channel. Fail safe/fail closed: flag-off and
pre-C2 PDBs decode to the empty map (sessions start fine), and a method whose
blobs do not decode is omitted rather than guessed.

Generation chaining: EmitDeltaForCompilation recomputes per-method occurrence
data from the fresh typed tree (computeRefreshedEncMethodDebugInfos, name-to-token
resolution fail closed on non-unique names) and chainEncMethodDebugInfos replaces
the updated methods' entries in the next-generation baseline, dropping entries the
fresh compile did not produce so stale data can never be matched. The delta PDB
does not yet re-emit EnC CDI rows; the in-memory chain is what C3 consumes, and
PDB persistence across session restarts is the documented remaining gap.
Add ClosureNameAllocator, the F# analogue of Roslyn's
EncVariableSlotAllocator.TryGetPreviousLambda/TryGetPreviousClosure
expressed over the C1 lambda occurrence model: a matched occurrence
(same two-pass LCS as the C1 diff, equal capture sets, recorded
baseline name) reuses the baseline closure class name verbatim; an
unmatched, capture-incompatible, or unmappable occurrence gets a fresh
generation-suffixed name ({base}@hotreload#g{gen}_o{ord}, the
DebugId(ordinal, generation) analogue); removed occurrences fall out of
the chain-forward table so their names are never reused.

The index-pair core of the C1 alignment is extracted into
TypedTreeDiff.alignLambdaOccurrenceIndexPairs and shared by both
consumers, so diff classification and name allocation can never
disagree about pairing. Pure data transformation: no IO, no IlxGen
state; covered by ClosureNameAllocatorTests (match/add/remove/nested/
capture-incompatible/fail-closed plus three-generation chaining).

Gates: service HotReload 365/365 (356 + 9 new), component HotReload
134/134.
Two layers of coverage for the C3 naming machinery:

- ClosureNameAllocatorTests now also drives the allocator over REAL C1
  extraction (checker compiles via the shared DiffTestHarness, made
  internal for reuse): adding a filter lambda yields a generation-2
  fresh name while the surviving map lambda reuses the baseline name,
  and generation 3 reuses both names from the chained table.

- New component ClosureIdentityTests pins the metadata-level invariant
  the delta path relies on today (and C4 extends): flag-on recompiles
  of an unchanged lambda set produce closure classes with identical
  names across three body-edit generations, so deltas can update the
  existing closure method bodies in place.

Gates: component HotReload 135/135, service HotReload 366/366,
EmittedIL 1212 passed / 3 skipped / 0 failed.
Bridge the C1 lambda occurrence model to IlxGen's closure naming seam so the
occurrence-keyed allocator can be wired into delta compiles:

- LambdaOccurrence gains RootExprStamp, the unique stamp of the occurrence's
  outermost Expr.Lambda (extraction bookkeeping, never part of the structural
  digest or alignment).
- GetIlxClosureFreeVars records stamp -> emitted closure type name into a new
  ConditionalWeakTable side channel (ClosureNameAllocationState, mirroring
  CompilerGeneratedNameMapState); recording is armed only by the emit hook for
  capture compiles, so flag-off output stays byte-identical.
- The fsc emit path joins the recording with the same tree's occurrence
  extraction (ClosureNameAllocator.computeBaselineClosureNameRows, fail closed
  on ambiguous compiled names and on members with incompletely recorded
  occurrences) and threads the per-method chain -> name tables through
  TryEmitWithArtifacts/CompilerEmitArtifacts into the baseline capture, where
  they are re-keyed by MethodDef token and stored as
  FSharpEmitBaseline.EncClosureNames alongside EncMethodDebugInfos.

Gates: compiler + both test projects build clean; component HotReload 136/136
(135 + new baseline-capture test), service HotReload 366/366, EmittedIL
1212/1212 (3 skipped).
Thread the C3 allocator into IlxGen lowering for session compiles:

- ICompilerEmitHook.PrepareForCodeGeneration now receives tcGlobals and the
  optimized impl files about to be lowered. When a session with baseline
  closure-name tables is active, the hook runs
  HotReloadBaseline.computeOccurrenceKeyedClosureNames (previous-generation
  occurrences vs fresh extraction, allocator per member with
  generation = session.CurrentGeneration) and installs the resulting
  stamp -> assigned-name table on the compiling CompilerGlobalState.
- The IlxGen closure call site consults the table FIRST and falls back to
  sequence replay; the replay-map slot is still consumed unconditionally so
  same-basename non-closure names keep their baseline replay positions.
  Surviving closures therefore reuse baseline class names verbatim even when
  the lambda set changes; added occurrences get {base}@hotreload#g{N}_o{i}.
- DeltaEmissionRequest.RefreshedClosureNameRows carries the allocator's
  refreshed per-method tables (recomputed deterministically at delta emission)
  and chainClosureNameRows chains them into the next-generation baseline with
  the chainEncMethodDebugInfos replace-or-drop semantics.
- Fail closed everywhere: no session, empty baseline tables, ambiguous
  compiled names, or unresolvable tokens leave lowering on pure sequence
  replay; flag-off compiles see a single failed weak-table lookup.

Gates: compiler + both test projects build clean; component HotReload 138/138,
service HotReload 366/366, EmittedIL 1212/1212 (3 skipped); the
multi-generation closure-identity and body-edit runtime tests now exercise the
allocator path live.
… in delta compiles

Component coverage for the C3 lowering wiring, driven through the session/hook
plumbing (flag-on compiles against an active session):

- three body-edit generations of a 2-lambda method keep closure class names
  byte-identical AND keep the chained session tables carrying the same names
  (the compiles now take the allocator path, not bare sequence replay);
- a recompile with a lambda ADDED IN FRONT of the survivors — the case
  sequence replay cannot handle — keeps every baseline closure name verbatim
  and gives the added occurrence the generation-suffixed
  {base}@hotreload#g1_o{i} name, which also chains forward for reuse.

Classification still rejects lambda-set changes at delta emission; emitting
the added member is Phase C4, documented in the design doc.

Gates: component HotReload 138/138, service HotReload 366/366.
Phase C4 sub-slice 1: writer + emitter support for ADDED TypeDef rows,
mirroring the Roslyn reference delta for "method gains its first capturing
lambda" (new display class). Reference recorded with a compiler-level
Compilation.EmitDifference harness (hot_reload_poc/src/csharp_enc_reference,
Roslyn 5.9.0) because the hotreload-utils workspace tool needs a net11-only
toolchain; the IDE pipeline ends in the same API, so the delta shape is
identical. Template captured verbatim in docs/hot-reload-member-additions.md.

Writer (FSharpDeltaMetadataWriter.emitWithTypeDefinitions):
- TypeDefinitionRowInfo / NestedClassRowInfo row models; TypeDef rows write
  FieldList/MethodList as 0 (Roslyn EnC parity - members are linked through
  the AddField/AddMethod EncLog pairs).
- EncLog: new TypeDef rows are plain Default entries placed before every
  AddField/AddMethod pair that names them as the parent; NestedClass rows
  are plain Default entries trailing the log. Both appear in EncMap.
- Serializer/SRM shadow writer/parity comparer extended for both tables.

Emitter (IlxDeltaEmitter):
- A fresh-compile type with the C3 allocator's generation-suffix marker
  ({base}@hotreload#g{N}_o{i}) and no baseline TypeDef token allocates the
  next delta TypeDef row; its fields/methods (including instance capture
  fields and the .ctor/Invoke pair) register as added members parented to
  the NEW row. Extends is remapped (TypeRef/TypeDef); TypeSpec base types
  rely on baseline passthrough and fail closed past the baseline table, as
  do generic closure classes (GenericParam rows unsupported) - both gaps
  documented.
- Added type tokens chain into the next-generation baseline TypeTokens and
  are reported in UpdatedTypeTokens (Roslyn ChangedTypes parity).

Tests: exact-EncLog writer test mirroring the C# template (service), plus
emitter tests for a hand-constructed nested closure type incl. mdv
validation (component). Component HotReload 140/140, service HotReload
367/367.
Phase C4 sub-slice 2: the payoff slice. A member that gains a lambda now
produces an applicable delta carrying the new closure class.

Classification (TypedTreeDiff): Added-only lambda sets become method-body
edits when the runtime advertises NewTypeDefinition+AddMethodToExistingType,
otherwise NotSupportedByRuntime naming the missing capability (C# parity).
Removed-only sets are allowed at Baseline capabilities (deleted lambda
bodies just become unreachable; the baseline closure class stays unused).
Capture-set changes stay rude.

Emission: the delta compile's fresh rewrite contains the C3 allocator's
{base}@hotreload#g{N}_o{i} closure class; the emitter selects it as an added
TypeDef (sub-slice 1 machinery), and generation-suffixed names are excluded
from basic-name alias buckets (they never alias a baseline closure). The
type token chains into the next-generation baseline, so gen N+1 body-edits
the added lambda in place.

Wiring fixes the runtime test forced:

- checker.StartHotReloadSession rebuilds the baseline from disk, which
  cannot carry EncClosureNames (CDI blobs have no name slots); it now
  carries the tables over from the in-process capture session it replaces
  when the MVID matches. Without this every watch-flow delta compile fell
  back to sequence-replay naming.
- MemberRef/TypeSpec token passthrough is now content-validated: an added
  lambda shifts the fresh compile's reference-row order, and positional
  passthrough silently swapped ListModule.Filter/Map MethodSpec bindings
  (BadImageFormatException at invoke). The baseline snapshots MemberRef row
  contents and TypeSpec blobs (ILBaselineReader, which also gained the
  missing InterfaceImpl row size - assemblies with interface impls had all
  later table offsets misread); the remapper validates positional reuse,
  falls back to content search, then appends (MemberRef) or fails closed
  (TypeSpec). Appended MemberRef rows chain forward for later generations.
- remapTypeSpecBlobWith: TypeSpec blobs are a bare Type (II.23.2.14), not a
  calling-convention-prefixed signature.
- The AsyncStateMachineAttribute heuristic now requires a MoveNext method,
  so closure classes sharing the {method}@hotreload naming no longer pick
  up a spurious attribute.

Runtime evidence (ApplyUpdate succeeds for added lambda creating a new
closure class): gen1 adds a third lambda in front of two survivors -> delta
EncLog carries the new TypeDef row + AddField/AddMethod pairs + NestedClass,
ApplyUpdate accepts, probe observes the new behavior; gen2 body-edits the
ADDED lambda -> method updates only, no new TypeDef rows.

Updated classification tests to the new semantics (added-without-capability
is NotSupportedByRuntime naming NewTypeDefinition; removed-only is a body
edit; async closure-chain shape changes follow the same gating).

Gates: component HotReload 141/141, service HotReload 367/367,
EmittedIL 1212/1212 (3 skipped). Both design docs updated.
Phase C4 sub-slice 3:

- Negative runtime gate: a capability-less session rejects an added lambda
  with NotSupportedByRuntime (FSHRDL016) naming NewTypeDefinition, before
  any emission.
- Removed-lambda runtime test: removing a lambda applies as a plain set of
  method-body updates (no TypeDef table entries; the baseline closure class
  stays, unused) and the new behavior takes effect - pinning the C#-parity
  allowance enabled in sub-slice 2.
- mdv/EncLog pattern-parity assertions on the real generation-1 delta
  against the recorded C# new-display-class template: the new TypeDef row's
  plain Default entry precedes its Add* entries, every AddField/AddMethod
  parent entry is immediately followed by the member row with the Default
  operation, the NestedClass row trails, and EncMap carries the TypeDef and
  NestedClass rows but never the Add* parent entries.

Gates: component HotReload 143/143, service HotReload 367/367.
… conduit

The C2 PDB writer wiring added the methodCustomDebugInfoRows option to
ilwritepdb.fsi; record the intentional drift and refresh the locked hashes.
Phase B2: instance fields added to existing CLASSES are emitted with the
same (TypeDef, AddField) + (Field, Default) EncLog pairing as the B1b
static path, validated against a recorded Roslyn EmitDifference reference
(csharp_enc_reference field_add scenario) and applied at runtime.

Classification: compareEntities now diffs the field segment structurally;
a pure field addition on a TFSharpClass is gated on the negotiated
AddInstanceFieldToExistingType/AddStaticFieldToExistingType capabilities
(RudeEditKind.NotSupportedByRuntime names the missing one) instead of
reporting TypeLayoutChange. Struct/record/union/enum layouts stay rude
permanently (runtime restriction, C# identical), enforced fail-closed in
the emitter via the fresh TypeDef's IsStructOrEnum.

Constructor pairing: a mutable class let binding folds its initializer
into the primary ctor, whose binding diffs as a plain MethodBody update;
ctor return-type identity is now void (the IL truth) so .ctor edits
resolve against baseline method tokens. [<DefaultValue>] val mutable
needs no pairing: the entity surfaces as a TypeDefinition edit and the
delta is a pure Field-row append - the metadata writer's empty-delta
short-circuit now keys on row payload, not method updates alone.

Runtime tests assert C# EnC semantics: existing instances read zeroed
values for the added field, new instances run the updated ctor and see
the initializer, and state survives multi-generation chains.
The runtime-identity signature gate flagged unchanged unit-returning members
(e.g. property setters) as a SignatureChange. diffImplementationFile snapshots the
baseline and updated trees with a single TcGlobals, so GetValReprTypeInCompiledForm's
isUnitTy-based "unit return -> None" mapping only fires for the g-native tree; the
other tree's unit return leaks through as Some unit and resolves to
NamedType("Microsoft.FSharp.Core.Unit") rather than VoidType, so the two sides compare
unequal even though the member is unchanged.

Fold the resolved unit identity back to void (the compiled IL truth: a unit-returning
F# method emits a void-returning method) so an unchanged unit-returning member yields
the same return identity on both sides of the diff.
…d check

The unresolved-method fail-closed check rejected edits whose only unresolved keys were
compiler-generated closures/companions (declaring type contains '@', e.g.
view@hotreload#g0_o0::Invoke), whose generation-specific names do not exist under the
same name in a fresh full recompile. That rejected supported computation-expression
edits with an UnsupportedEdit.

Fail closed only on genuinely-requested user methods that fail to resolve; tolerate
compiler-generated companions (the historical drop behavior) since the user-facing edit
is still emitted through the owning member. Update the emitDelta unit test to assert the
fail-closed throw for an unresolved user method.
NatElkins added a commit to NatElkins/fsharp that referenced this pull request Jul 1, 2026
…tors

Compiler-generated occurrence names (name@line-N) are allocated from process-wide
counters on CompilerGlobalState that accumulate across compilations. When a warm
checker re-emits the same project in-process, an unchanged closure therefore gets a
different occurrence suffix than the previous emit, so consumers that align generated
names across compilations (Edit-and-Continue delta emission, dotnet#19941)
cannot match them.

Add an internal ResetCompilerGeneratedNameState to NiceNameGenerator (clears the
per-(name, file) occurrence counters), StableNiceNameGenerator (clears the cached
stable names and the inner counters), and an aggregate on CompilerGlobalState that
resets all three generators, restoring the fresh-process name layout. Callers must
ensure no compilation is concurrently generating names.

No in-tree caller yet; the consumer is the hot reload emit path in dotnet#19941.
Covered by unit tests proving drift without reset, exact replay after reset, and that
the stable-name cache itself is cleared.
NatElkins added a commit to NatElkins/fsharp that referenced this pull request Jul 1, 2026
…ethod CDI emission

Adds an internal AbstractIL module implementing, byte for byte, the three Portable PDB
CustomDebugInformation blob formats Roslyn persists per method for Edit and Continue
(EnC Local Slot Map, EnC Lambda and Closure Map, EnC State Machine State Map), with
serializers, deserializers, a portable PDB read-back helper, and an occurrence-key
packing helper for deterministic syntax-offset slots.

Plumbs an optional methodCustomDebugInfoRows side channel through the IL binary writer
options into the portable PDB generator so a compilation can attach CDI rows to named
methods. Names that do not identify exactly one method row are dropped. All existing
writer call sites pass an empty map, so emitted PDBs are byte-identical to before.

No in-tree caller populates the map yet; the consumer is the F# hot reload work in
dotnet#19941, following the same pattern as dotnet#20017 (land isolated, test-covered
infrastructure first, wire the feature later).

Tests: blob round-trips, Roslyn golden-byte encodings, cross-validation against
CDI blobs emitted by a real Roslyn compilation, fail-closed occurrence-key packing
(including an int32-overflow regression where a wrapped negative key previously
escaped the bound check), and end-to-end synthetic PDB emission proving correct
MethodDef parenting, zero rows for an empty map, and no rows for absent or
ambiguous names.
NatElkins added a commit to NatElkins/fsharp that referenced this pull request Jul 1, 2026
Adds an internal, standalone ECMA-335 Edit-and-Continue metadata delta writer to
AbstractIL: delta #- table stream and heap construction (DeltaMetadataTables,
DeltaMetadataSerializer, DeltaTableLayout, DeltaIndexSizing), ECMA-335 II.24.2.6
coded-index encoding (DeltaMetadataEncoding), EncLog/EncMap emission, generation GUID
chaining, user-string and standalone-signature token calculators (IlxDeltaStreams),
and the coordinating writer (FSharpDeltaMetadataWriter) over a plain row-description
input model (DeltaMetadataTypes, ILDeltaHandles, ILMetadataHeaps).

The writer's inputs are row records (names, tokens, signatures, RVAs) plus heap
offsets; it has no dependency on any semantic diffing or session machinery. It
compiles with no in-tree consumer by design: the consumer is the F# hot reload work
in dotnet#19941, following the same upstreaming pattern as dotnet#20017 and dotnet#20018
(land isolated, test-covered infrastructure first, wire the feature in a later PR).

One line of ilwrite.fsi is touched to expose the pre-existing markerForUnicodeBytes
so the delta writer reuses the exact string-marker logic of the full writer. No
behavior change for any existing code path.

Tests (130): coded-index encodings asserted against the production definitions and
ECMA-335 II.24.2.6 order, System.Reflection.Metadata reader parity over emitted
deltas, EncLog/EncMap correctness, stream layout, heap and index sizing,
multi-generation heap-offset chaining asserted against computed expected values,
standalone-signature rows asserted at baseline+1 from a real seeded baseline, and
serializer failure paths.
NatElkins added 2 commits July 1, 2026 18:52
…naming

The delta emitter's fail-closed guard for synthesized types with no baseline
counterpart only recognized the stable '@hotreload' naming scheme. External and
flag-off builds name closures with the fresh-fsc '@<line>' scheme (for example
'Pipe #1 stage #3 at line 16@16' from debug pipe desugaring), so an added
pipe-stage closure skipped the guard, fell through to a baseline token-map lookup
known to be invalid, and crashed EmitDelta with an ilwrite InternalError surfaced
as DeltaEmissionFailed. dotnet-watch treats UnsupportedEdit as a graceful
rebuild-and-restart but DeltaEmissionFailed as an error, so the crash broke the
watch experience for a common edit.

Broaden the guard to IsCompilerGeneratedName, covering both naming schemes.
Legitimately added lambdas are unaffected: generation-suffixed added closures
resolve through the earlier added-TypeDef path, and added user types resolve
through addedUserTypeNames before the guard.

Verified with a full external-build repro (multi-module app, pipe stage inserted
before existing stages): EmitDelta now returns the FSHRDL099 closure-chain rude
edit instead of crashing. HotReload component suite 229 passed, service suite
413 passed. A new disk-started flag-off contract test pins the rejection shape
for this topology; with a flag-on baseline the same insertion remains supported
via CDI-reconstructed occurrence keys.
Per-edit hot reload currently requires an external 'dotnet build -t:Compile' to
refresh the obj assembly the delta emitter reads, making every edit pay a full
MSBuild+fsc invocation on top of the in-process check (measured ~7s/edit vs
~4.7s for a plain build). This adds an experimental, flag-gated path that
produces the same on-disk obj DLL and PDB in-process from the checked project
the session already has, collapsing the two compiles into one (spike-measured
~3.0s/edit with a correct one-method delta).

- FSharpChecker.CompileFromCheckedProject (internal): emits the assembly from
  cached typecheck results (finalize CCU, dedupe QualifiedNameOfFile, optimize,
  IlxGen, write DLL + portable PDB to the requested path), adapted from the
  dotnet#19267 prototype. Before codegen it clears leaked hot-reload
  closure-name state and resets the compiler-generated-name counters
  (ResetCompilerGeneratedNameState) so the emit lays out generated names exactly
  like a fresh-process build; without this an unchanged closure drifts to a new
  occurrence suffix and the delta mapping degenerates.
- FSharpCheckProjectResults.CompilationData and TcImports.NormalizeAssemblyRef
  (internal) supply the compilation inputs, from the same prototype.
- ResetCompilerGeneratedNameState on the name generators, matching
  dotnet#20017 so the branch converges with main when that PR lands.
- Session wiring: when FSHARP_HOTRELOAD_INPROCESS_COMPILE is truthy,
  EmitHotReloadDelta runs the in-process compile after session-active
  validation and before the output freshness pipeline, failing closed as
  DeltaEmissionFailed on any compile error. With the flag unset the code path
  is unchanged beyond a single environment read. The external build remains
  the default; dotnet-watch integration opts in separately.
- The emitted portable PDB keeps the sibling-PDB freshness contract: sequence
  points for line-shifted, unedited methods come from the new compile, not a
  stale external-build PDB (regression-tested; the test fails without the PDB
  emission).

Verified: service HotReload suite 414 passed 0 failed (including the new
flag on/off contract test asserting one updated method with no external
rebuild, a +1 LineUpdate for an unedited shifted method, and the unchanged
stale-output refusal with the flag off); component HotReload suite 229 passed
0 failed (flag-off neutrality).
NatElkins added a commit to NatElkins/fsharp that referenced this pull request Jul 2, 2026
…tors

Compiler-generated occurrence names (name@line-N) are allocated from process-wide
counters on CompilerGlobalState that accumulate across compilations. When a warm
checker re-emits the same project in-process, an unchanged closure therefore gets a
different occurrence suffix than the previous emit, so consumers that align generated
names across compilations (Edit-and-Continue delta emission, dotnet#19941)
cannot match them.

Add an internal ResetCompilerGeneratedNameState to NiceNameGenerator (clears the
per-(name, file) occurrence counters), StableNiceNameGenerator (clears the cached
stable names and the inner counters), and an aggregate on CompilerGlobalState that
resets all three generators, restoring the fresh-process name layout. Callers must
ensure no compilation is concurrently generating names.

No in-tree caller yet; the consumer is the hot reload emit path in dotnet#19941.
Covered by unit tests proving drift without reset, exact replay after reset, and that
the stable-name cache itself is cleared.
NatElkins added 12 commits July 1, 2026 20:48
… path

The flag-gated in-process compile wrote the assembly to disk and the session then
re-parsed the same file it had just written, behind two stable-file polls that only
exist for external writers. CompileFromCheckedProject now emits once in memory,
persists the DLL and portable PDB bytes to disk (the runtime and debugger contract
is unchanged), parses the bytes back, and returns the module; the session consumes
it directly and skips the stable-file wait and the on-disk re-parse under the flag.

The module handed over is the READ-BACK representation, parsed from the emitted
bytes with the assembly refs the writer normalized during serialization. Passing
the writer-side module instead is not equivalent: the delta emitter's symbol
matcher failed to resolve updated methods against it, and its scope refs differ
from the on-disk bytes that seed the baseline token mapping.

Flag-off behavior is unchanged: external outputs still go through the stable-file
wait and disk read. Output fingerprints are still computed from the on-disk bytes
either way, so the stale-output contract and commit chain are untouched.

Verified: service HotReload suite 414 passed 0 failed (the flag on/off contract
test now exercises the in-memory handoff), component HotReload suite 229 passed
0 failed.
BaselineMetadataReader's small members (TypeRefCount and friends) get cross-module
inlined in Release builds, and the inlined code references the module-private
MetadataContext record directly. The CLR then rejects the field access at runtime
('Attempt by method ... to access field ... RowCounts@ failed') when the inlined
copy lives in another file's state machine, which broke hot reload session start
entirely on Release FSharp.Compiler.Service: every edit silently fell back to
rebuild and restart. Debug builds never inline these members, which is why the
suites stayed green.

Make MetadataContext internal instead of private: same-assembly access is what the
inliner needs, and the type remains hidden outside the assembly.

Measured with the fix on a 72-file app under dotnet watch (Release FCS, in-process
compile flag on): median ~1.6s per edit applied in place with state preserved,
versus ~1.9-2.6s for a plain per-edit dotnet build on the same machine. Debug
suites unchanged: service HotReload 414 passed, component 229 passed.
computeSymbolChanges diffed every file's baseline tree against the fresh tree on
every edit, making the diff cost proportional to project size rather than to the
edit. When the TransparentCompiler returns the same CheckedImplFile instance the
committed baseline holds, nothing in that file can have changed, so the pair is
skipped. Reference identity is the sound gate here: source-text equality is not,
because F# type inference propagates across files and a downstream file's typed
tree can change without its source changing (such files come back as fresh
instances and are still diffed).

Covered by a session test proving unchanged earlier files are reference-equal
between the committed baseline and the fresh check while the edited file is not.
Trace under FSHARP_HOTRELOAD_TRACE_METHODS reports skipped vs diffed counts.
Service HotReload suite: 415 passed, 0 failed.
…iles

computeRefreshedEncMethodDebugInfos and computeOccurrenceKeyedClosureNames walked
every implementation file on every edit, even though EmitDelta chains their rows
only for the updated method tokens. Both now accept an ImplementationFileScope:
ReferenceChanged pairs the committed baseline files with the fresh check's files
by qualified name and skips pairs whose CheckedImplFile instances are reference
equal, the same sound gate the typed-tree diff uses (source-text equality is not
sound under cross-file type inference; reference identity is).

Fail-open everywhere: duplicate file keys, added, removed, or renamed files, or a
missing committed baseline fall back to the full walk. The codegen-time closure
allocator install keeps the full walk; only the delta-emission refresh is scoped.

Name-collision note: member-occurrence keying is fail-closed on globally unique
compiled names. Under scoping, a fresh-only collision between a changed and an
unchanged file would not be detected as ambiguous, but the binding still cannot
misfire: baseline token lookup stays baseline-global (duplicates are excluded
there), and refreshed rows are only consumed for updated method tokens, which an
unchanged file cannot produce.

Service HotReload suite: 415 passed, 0 failed. Component HotReload suite: 229
passed, 0 failed.
The in-process compile re-optimized every implementation file on every edit. At
the dev-loop settings this path always uses (--optimize- with minimal passes),
per-file optimization from a fresh environment is byte-identical to the threaded
batch fold - verified by an A/B experiment serializing both variants of a 12-file
stress project (shared anonymous records, literals, ordered dependencies) and a
73-file application to identical SHA256s. Whole-assembly IlxGen is unchanged: a
prior experiment falsified per-file codegen (assembly-canonical anonymous types,
last-file entrypoint semantics, shared PrivateImplementationDetails).

Under FSHARP_HOTRELOAD_INCREMENTAL_EMIT, each file's optimized tree is cached in
a ConditionalWeakTable keyed by the input CheckedImplFile reference: the checker
returns reference-identical instances for unchanged files, so a hit is exactly
what re-optimizing would produce, and the table's lifetime rides the checker's
own typed-tree cache. Misses optimize that file alone from the fresh environment.
Any failure falls open to the threaded batch path; with the flag unset the
existing path runs untouched.

Nothing accumulated by the batch fold is consumed downstream on this emit path:
the optimizer info is discarded and the optimization-data resource list is empty,
so inputs to codegen and module creation are identical apart from the cached work.

Tests: flag-on bytes equal flag-off bytes after a one-file edit against a warmed
cache (pins the independence result); a session with both flags applies two
successive edits with exactly one updated method each (cross-generation splice).
Service HotReload suite: 417 passed, 0 failed. Component: 229 passed, 0 failed.
Per flag-on edit the fresh module was serialized twice: once by the in-process
compile writing the output assembly, and again inside the delta emitter, which
re-serialized the whole module in memory purely to obtain PE bytes, a PDB, and
token mappings. The session also re-read the freshly written PDB from disk.

CompileFromCheckedProject now returns the emitted assembly bytes, PDB bytes, and
token mappings alongside the parsed module, and the session threads them into the
delta emission request. The emitter consumes them in place of its re-serialization;
absent artifacts (the external-build path and the low-level test entry points) keep
the existing rewrite, and the sibling-PDB disk read remains the fallback when no
PDB bytes were threaded.

The writer's own token-mapping closures are not substitutable here: they are keyed
by writer-side signatures and can throw when invoked with read-back IL objects.
The threaded mappings are instead built from the read-back objects' metadata row
indices, which come from parsing the exact emitted bytes, so tokens correspond to
the same bytes the emitter's PEReader consumes. The threaded PDB is strictly
better than the rewrite's, which carries no sequence points for disk-read modules.

No delta byte-equality A/B test: each emit mints a fresh EncId by design, so
equivalent emissions are never metadata-byte-identical; coverage comes from the
extended session test asserting the delta PDB parses with method debug information
on a line-shift edit, plus both suites. Service HotReload: 417 passed, 0 failed.
Component HotReload: 229 passed, 0 failed.
Per-edit latency work so far has relied on ad-hoc instrumentation (temporary
printfn patches or external log scraping), which cannot decompose a live watch
session and keeps getting rewritten. This adds a permanent trace under
FSHARP_HOTRELOAD_TRACE_TIMING, following the existing FSHARP_HOTRELOAD_TRACE_*
lazy-flag pattern, printing one '[fsharp-hotreload][timing] <stage>=<N>ms' line
per stage.

Session stages (EmitHotReloadDelta): parseAndCheck, inProcessCompile (when the
in-process flag runs it), staleCheck, readIlModule (when the disk path runs),
emitDelta, and total. Emission stages (EmitDeltaForCompilation): symbolChanges,
buildRequest (symbol-to-token mapping through the refreshed EnC and closure-name
tables), and emitCore.

The flag is read once lazily; with it unset no stopwatch is created and the only
cost is a boolean check per call. The timed bodies are wrapped by two small
combinators (timeAsync/timeSync) rather than inline stopwatch plumbing, which
re-indents the emission body but does not reorder or alter any operation; the
full HotReload service suite (417) and component suite (229) pass unchanged.

Motivation: decomposing the remaining per-edit cost between the FCS stages and
the dotnet-watch pipeline decides whether further work targets incremental
codegen (per-file IlxGen has five documented cross-file couplings to solve) or
watch-side overhead. That decision needs stage numbers from real watch sessions,
not estimates.
FSHARP_HOTRELOAD_DUMP_DELTAS=<dir> persists each emitted delta's metadata, IL,
and PDB blobs plus the updated token list, so a live watch session's output can
be decoded offline. Best-effort writes; unset means no behavior change. This was
built to investigate a suspected stale-string bug on a real Giraffe app, where it
let the actual session delta be decoded and proven correct (the report was a
benchmark polling an endpoint the edited handler never served): correct method
row in the EncLog, and the updated body's ldstr operand resolving to the appended
delta user-string entry carrying the new literal.

Also adds a session test decoding an emitted delta's IL to assert the ldstr
operand points past the baseline user-string heap into the delta's appended
entry with the edited string, pinning the absolute-offset contract for delta
user strings against a Giraffe-shaped module (eta-expanded function-typed
binding among sibling closures). Service HotReload suite passes.
…ed baselines

The in-process hot reload compile cleared all closure naming state and emitted
line-based closure names. Against a flag-on baseline (--test:HotReloadDeltas,
stable occurrence-derived names plus EnC CDI) those names mis-map: closure-bearing
methods hit the ambiguous-synthesized-mapping rude edit, so any project opting
into stable baselines lost the in-process fast path entirely. Meanwhile flag-off
baselines suffer the line-index problem: a line-adding edit renames every closure
below it and falls back to restart, which on closure-dense real-world files (a
Giraffe endpoints list) makes restarts the common case.

CompileFromCheckedProject now takes a naming mode. Clear mode is the existing
behavior for line-based baselines. Preserve mode mirrors the fsc emit hook's
PrepareForCodeGeneration discipline: the session installs the project's
synthesized-name map, and the compile computes the stamp-keyed closure-name
replay over the OPTIMIZED implementation it is about to lower (checked-tree
stamps are not the stamps IlxGen consults; computing over optimizedImpls matches
the hook and was required for the replay to hit) and installs it via
ClosureNameAllocationState before IlxGen. The session selects the mode from the
baseline itself: no EncClosureNames means a line-based baseline and clear mode,
otherwise preserve. Cleanup runs in a finally exactly like the hook's.

Acceptance test: flag-on baseline, in-process compile, a line-adding edit above
sibling module-level lambdas emits Ok with exactly one user-authored updated
method; on the previous code this shape failed closed as FSHRDL099. Service
HotReload suite: 420 passed, 0 failed. Component suite: 229 passed, 0 failed.
Two synthesized-closure families could not be aligned across compiles after a
line-adding edit, so such edits always fell back to restart on closure-dense
real-world files (reproduced on Giraffe's endpoint list, failing identically on
the external and in-process paths):
- debug-pipe closures embed the source line in the basic name itself
  ('Pipe #1 stage #2 at line 28'), so a line shift changes the name and the
  fresh type has no baseline counterpart;
- value-binding list closures use sequential ordinals (endpoints@hotreload-N)
  allocated in encounter order, so shifts make one fresh name match several
  baseline candidates.

Add a last-resort positional pairing to the delta emitter's synthesized-type
match ladder, consulted only after exact and path-normalized matching fail or
tie. Groups are keyed by enclosing type plus a line-normalized basic name; the
normalizer recognizes only the known generated forms (pipe labels, @hotreload
ordinals, legacy @line ordinals) and refuses everything else, including
occurrence-keyed names, which already match exactly. Pairing requires equal
group counts, distinct ordinals on both sides, an unused baseline token, and a
structural shape match per pair (generic arity, base type, interfaces, field
type multiset, method name/arity set, with self-references normalized), and
produces a bijection; any failure keeps today's fail-closed rude edit, so a
structural change such as adding a pipe stage or list lambda still requires a
rebuild rather than risking a mispair, which would be silent corruption.
Baseline synthesized-type shapes are derived from the module at session start
and chained for delta-added types; no persisted format changes.

This is the recovery layer: it makes line-shift edits on these families apply
in place when the replayed names drift. A follow-up addresses the drift at its
source by keying the replay name map on line-normalized identity so matched
closures keep their generation-0 names, Roslyn's model; exact matching then
succeeds without inference and this pairing remains a guarded fallback.

Component acceptance: in-process and disk-started line-shifted pipe closures
and ordinal-shifted value-list closures each emit with exactly one user-authored
updated method; count-mismatch cases stay UnsupportedEdit. Service suite 423
passed, component 234 passed (2 expected manual-host skips).
Replay of compiler-generated closure names was keyed by the raw basic name, and
debug-pipe closures embed the source line in the basic name itself ('Pipe #1
stage #2 at line 28'), so any line-adding edit changed the bucket key, missed
the recorded generation-0 allocation, and produced fresh names the baseline
could not match. Value-binding list closures shared a bucket whose sequential
ordinals depended on encounter order and drifted similarly. Both families then
depended on the positional-pairing recovery layer or fell back to restart.

Key replay buckets by the line-normalized basic name and keep the original
generation-0 full names as the values, so a matched closure whose code moves
from line 28 to line 30 is handed back its line-28 birth name. This is Roslyn's
Edit and Continue model: synthesized-member identity is established at first
allocation and replayed exactly, making downstream matching exact rather than
inferred. The normalization grammar is extracted to one shared home in
GeneratedNames and reused by the map, the baseline snapshot collector, the
symbol matcher, and the delta emitter's pairing layer, which now acts purely as
a recovery net behind exact matching.

Snapshot compatibility is handled at load: raw legacy keys are normalized,
duplicate buckets merge, replay buckets canonicalize by ordinal, and gapped
buckets recover the unambiguous raw birth basic name from recorded entries
rather than synthesizing a name that would lose the original line text (a
normalized-key fallback remains for genuinely ambiguous recovery). The
ICompilerGeneratedNameMap surface and the persisted Map<string, string[]>
snapshot shape are unchanged; behavior without an installed map is untouched.

Acceptance: the flag-on in-process session test decodes TypeDef names from the
baseline and fresh assemblies after a line-adding edit and asserts exact name
equality for the unchanged closure families. Service HotReload suite: 426
passed, 0 failed. Component suite: 234 passed, 0 failed, 2 expected skips.
…cation

Value bindings like an EndpointRouting endpoints list produce MIXED closure
allocation: some closures occurrence-keyed, others sequence-replay slots with
consume-then-override gaps. The occurrence names for such a binding may not be
covered by any reconstructed EncClosureNames table (those tables cover method
owners like handlers), but the synthesized-name snapshot preserves the complete
consumed-slot layout, including the generation-0 occurrence names, and is
sufficient to replay the mixed allocation exactly.

The in-process compile's naming-state install was gated on EncClosureNames
being non-empty alone. A baseline whose occurrence coverage lives only in the
snapshot then compiled in clear mode, consumed sequence slot 0 as a plain
replay name where the baseline had an occurrence override, and the divergent
name failed the synthesized-type mapping (gracefully, as a rude edit and
restart). Widen the gate: install the snapshot when EncClosureNames is
non-empty OR when the snapshot itself contains occurrence-keyed closure names,
the direct evidence that mixed slot consumption exists. Pure replay snapshots
without occurrence names remain insufficient evidence, since broad installation
for empty-table baselines risks unsafe slot filling; that shape keeps the
current fail-closed behavior, which is also the old-baseline grace path.

No CDI, snapshot format, or API changes. Acceptance: a reduced EndpointRouting
shaped mixed bucket, verified by decoding baseline PE TypeDefs to show both
naming families, applies a line-adding edit with exactly one user-authored
updated method and byte-equal endpoint closure names in BOTH the in-memory
capture and disk-started topologies; the pure-replay empty-table shape still
fails closed. Service HotReload suite: 428 passed, 0 failed. Component suite:
234 passed, 0 failed, 2 expected skips.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: New

Development

Successfully merging this pull request may close these issues.

6 participants